A Formal Semantic Analysis of Gesture

This paper presents a formal semantics for productive iconic and deictic gestures that speakers use in tandem with speech in face-to-face conversation. In contrast to conventionalised, emblematic gestures, the meaning that productive gestures derive from their form is incomplete, and their content can be determined only through links to context, including previous gestures and simultaneous speech. Our model describes both the meaning revealed by form and the more specific interpretation in context. We build on established models of discourse to capture key insights from the descriptive literature on gesture—for instance, that synchronous speech and iconic gesture express a single thought. Our theory exploits dynamic semantics and rhetorical connections among communicative actions to describe how context contributes to the interpretation of gesture, including constraints on co-reference across speech and gesture, and across sequences of gesture.